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Abstract 


In 2007, the Washington State Board for Community and Technical Colleges 
(SBCTC) launched a performance reporting and funding policy called the Student 
Achievement Initiative (SAI) both to improve public accountability by more accurately 
describing what students achieve from enrolling in community colleges and to provide 
incentives to colleges through financial rewards for increasing student success. This report 
presents findings from a three-year evaluation of the initiative that was designed to assess 
how and to what extent the SAI model of performance funding encourages colleges to track 
trends in student achievement and improve student outcomes. The quantitative component of 
the evaluation was an analysis of “achievement point” accumulation by colleges over the 
period 2007 to 201 1 . The qualitative component was based on a synthesis of approximately 
250 interviews with faculty, staff, and administrators at 20 of Washington State’s 34 
community and technical colleges that took place in spring 2012. 

Key broad findings include: 

• The SAI is viewed as one force among others pushing the colleges to improve student 
success. The funding is not a significant factor motivating the colleges, largely 
because the amount (less than one percent of the system’s total operating budget) is 
too small to have much impact. 

• On average, the colleges increased their point total by 31 percent between 2007 
and 2011, with the relative positions of the colleges remaining stable. Although 
there was evidence of some gains in momentum (i.e., forward progress) for students 
who were already accumulating credits and making progress, overall student 
momentum does not seem to have changed much during the period in which the SAI 
has been in effect, even as aggregate achievement points have increased. 

• While larger colleges earn more awards than smaller colleges, there is little evidence 
that colleges serving more at-risk, low-income students are penalized by the SAI 
awards method. Consistent with the SAI’s goals, the basic skills metric appears to 
have encouraged enrollment from traditionally underserved groups. 

• The intermediate milestone framework is viewed as a helpful way to focus 
collective efforts on student progression and publicly account for college 
performance. In order to understand the impact of strategies for improving student 
outcomes, however, colleges have found they need to use longitudinal cohort data in 
conjunction with the cross-sectional SAI metrics. The funding mechanism has 
proved problematic and unpopular, as SAI funding has come from reallocated base 
funds rather than as additional funds as originally intended. 



1. Introduction 


In 2007, the Washington State Board for Community and Technical Colleges 
(SBCTC) launched a performance reporting and funding policy called the Student 
Achievement Initiative (SAI). The purpose of the SAI is both to improve public 
accountability by more accurately describing what students achieve from enrolling in 
community colleges and to provide incentives to colleges through financial rewards for 
increasing the levels of achievement attained by their students. A set of principles, 
developed by a task force, served to guide the specific design and implementation of the 
initiative (see Box 1). 


Box 1 

Principles for the Student Achievement Initiative 

Overall Principles 

• The initiative leads to improved educational attainment for students, specifically 
the goal of reaching the "tipping point" and beyond. 

• The initiative allows colleges sufficient flexibility to improve student achievement 
according to their local needs. 

• The initiative results in the identification and implementation of successful 
practices to improve student achievement system-wide. 

Principles for Measurement 

• Performance measures recognize students in all mission areas and 
reflect the needs of the diverse communities served by colleges. 

• Performance measures must measure incremental gains in students' educational 
progress irrespective of mission area. 

• Measures are simple, understandable, and reliable and valid points in students' 
educational progress. 

• Measures focus on student achievement improvements that can be influenced by 
colleges. 

Principles for Funding 

• Colleges are rewarded for improvements in student achievement. 

• Funding is structured so that colleges compete against themselves for continuous 
improvement rather than competing with each other. 

• Funding is stable and predictable, and cumulative over time. 

• Incentive funding rewards student success and becomes a resource for adopting 
and expanding practices leading to further success. 

• New funds provide the greatest incentive. 
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The initiative makes use of a framework for tracking the number of “achievement 
points” attained by students enrolled in the state’s 34 public two-year colleges. Colleges 
earn points when students achieve one or more educational milestones, which are 
organized along a continuum from remedial programs (including adult basic education 
and pre-college developmental education) up through the completion of credentials and 
training programs. Colleges receive funding for increases in total achievement points 
attained by students in a given year compared to the prior year. Colleges earn 
achievement points for milestone attainment in four areas, as shown in Table 1. 


Table 1 

Student Achievement Initiative Achievement Points 


Milestone Area 


Achievement Points 

Building toward college-level skills 

• 

Basic skills gains— increase in skill level based on a standardized 
test 


• 

Passing a pre-college writing or math course that would qualify 
student to advance to the next level 

First-year retention and progress 

• 

Earning 15 quarter credits of college-level coursework 


• 

Earning 30 quarter credits of college-level coursework 

Completing college-level math 

• 

Passing a math course required for either technical or academic 
associate degree 

Completions 

• 

Completing a degree, occupational certificate, or apprenticeship 
training 


Funding for increased points is determined by the amount of dollars available to 
fund the initiative and the total point gains achieved across the system. The SAI funding 
mechanism does not affect the formula by which the bulk of the system’s budget is 
allocated to colleges. SAI funding is provided as a financial reward in addition to the 
amount each college receives through the system’s basic funding mechanism. Once 
earned, a college’s reward is added to its base budget. However, due to budget reductions 
imposed on the college system nearly every year since the inception of the SAI, college 
base budgets have been reduced to form the new pool of SAI funds each year. 
Consequently, SAI funds earned in one year have not in fact increased college base 
budgets in subsequent years, as was the original intention. The SAI has continued to offer 
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the chance for colleges to earn funds on top of base funding, but these funds have been on 
top of a shrinking base budget. To date, annual SAI funding has constituted less than one 
percent of the system’s total operating budget. 

The Community College Research Center (CCRC) at Teacher’s College, 
Columbia University, in partnership with the Institute for Higher Education Leadership & 
Policy (IHELP) at Sacramento State University, conducted a three-year evaluation to 
assess the implications of the SAI performance funding policy for state community 
college performance incentive policies nationwide. The study, funded by the Bill & 
Melinda Gates Foundation, was designed to assess how and to what extent the SAI model 
of performance funding encourages colleges to improve student outcomes, and to track 
trends in student achievement during the implementation period. Findings from the study 
are intended to inform the SBCTC, as it considers revisions to the initiative, 1 and to guide 
efforts by other states to develop more effective funding mechanisms for community 
colleges. 

The research questions framing the study were: 

1) To what extent does the prospect of performance funding and 
the availability of performance data motivate colleges to 
implement policies and practices intended to improve student 
outcomes? 

2) What is the range of performance in Washington’s community 
and technical colleges according to the achievement point 
measures, and what are the trends in college performance over 
time? 

3) What distinguishes colleges that perform well according to the 
SAI framework from those that do less well? 

4) Is the SAI sustainable? 

In spring 2010, CCRC analyzed the points accumulated by the colleges during the 
two-year period from 2006-07 through 2008-09. The analysis examined the change in 


1 In 2011, the Washington Association of Community and Technical Colleges (WACTC), the organization 
of community and technical college presidents that develops policy recommendations to the SBCTC and to 
the system, formed the Student Achievement Advisory Committee to advise WACTC in making its 
recommendations to the State Board on changes to the SAI metrics and funding mechanism. Final 
recommendations have not been presented to the SBCTC as of the writing of this report. 
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total points, points by individual metric, and points per student, and it also explored the 
effect of individual student characteristics and college characteristics on these various 
performance indicators. 

In addition, CCRC-IHELP researchers conducted interviews with personnel at 17 
of the 34 Washington State community and technical colleges. The 17 colleges were the 
same colleges that were involved in the formative evaluation CCRC conducted of the 
SAI during the 2007-08 “learning year,” before performance funding was provided to 
colleges (Jenkins, Ellwein, & Boswell, 2009). In both 2008 and 2010, interviews were 
conducted at eight colleges through site visits and at nine colleges via telephone. The 17 
colleges were chosen to be representative of the 34 Washington State community and 
technical colleges in terms of size and location (i.e., urban, suburban, or rural). Technical 
colleges as well as comprehensive community colleges were included. The 2010 research 
produced an internal report for the SBCTC as well as a published policy brief (Shulock & 
Jenkins, 2011) that discussed the three overriding issues that the SBCTC and the state’s 
two-year colleges have been addressing over the last several years as they continue to 
implement the initiative: designing a valid measurement system, designing an effective 
performance funding system, and fostering conditions at the colleges to use the data to 
improve performance. 

This report revisits the four general research questions about the operation and 
sustainability of the Student Achievement Initiative based on more recent research. As in 
our earlier research, the analysis included a quantitative and a qualitative component. The 
quantitative component was an analysis of achievement point accumulation by colleges 
over the period 2007 to 2011. It was intended to address research questions (2) and (3) by 
analyzing patterns of achievement point gain across the colleges and identifying any 
correlations between achievement points and college characteristics. The quantitative 
analysis also considered the robustness of the specific measurement system by analyzing 
whether alternative measures of performance would have yielded different funding 
allocations. Finally, it examined whether, as achievement points have increased over 
time, there is evidence that more students are progressing further toward completion of 
postsecondary credentials, i.e., whether individual student momentum has increased since 
the SAI was implemented. 
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The qualitative component was a synthesis of approximately 250 interviews with 
faculty, staff, and administrators at 20 of the 34 colleges, through site visits to eight 
colleges and phone interviews at the other 12, conducted in spring 2012. The 20 colleges 
were the original 17 revisited and three colleges added to ensure inclusion of the highest 
performers, in terms of increase in total points and increase in points per student. The 
qualitative research in this final year of the three-year study addressed research questions 
(1), (3), and (4) by examining awareness of and opinions about SAI metrics and funding, 
the use of SAI data, and the impact that the SAI has had on colleges’ understanding of, 
and efforts to increase, student achievement. 

The remainder of this report will present the findings from both the quantitative 
and qualitative components of the research. It begins with an overall summary of 
findings, followed by sections that address — integrating the qualitative and quantitative 
findings — each of the four overall research questions. A concluding section summarizes 
our findings with respect to the original design principles and discusses how well those 
principles have been followed and how well they have served the goal of the initiative. 


2. Summary of Findings 


Research Question 1 

To what extent does the prospect of performance funding and the availability of 
performance data motivate colleges to implement policies and practices intended to 
improve student outcomes? 

After five years of implementation of the SAI, overall awareness of the initiative 
across the colleges is low, although in many cases low awareness can signal 
institutionalization of broader student success efforts. Knowledge of the initiative is still 
mostly limited to administrators and those faculty and staff who serve on committees 
overseeing activities related to student success or who receive SAI funds for related 
projects. A culture of evidence, in which colleges are using data and research for 
decisions on how to improve student outcomes, is developing across the colleges, but it is 
in the early stages at most of them and is only partly attributable to the SAI, with other 
initiatives and new accreditation requirements having a substantial impact. Most 
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respondents across the colleges acknowledge a change in culture in recent years, with an 
increased focus on student success. The SAI is seen as just one of many forces pushing 
colleges to improve student success. SAI funding is not a significant factor motivating the 
colleges, largely because the amount is too small to have much impact. Moreover, due to 
cuts in state funding that have occurred since the SAI was initiated, the funding has not 
been a “bonus” based on new money, as originally envisioned, but rather has been a 
reallocation of the overall system base budget, which declined substantially during this 
period. Most respondents view SAI dollars as a source of funding for small-scale projects 
rather than as motivation for the college as a whole to change in systemic ways to 
improve student success. 

Research Question 2 

What is the range of performance in Washington ’s community and technical colleges 
according to the achievement point measures, and what are the trends in college 
performance over time? 

On average, the colleges increased their point total by 31 percent between the 
baseline year (2007) and 2011. Based on our quantitative analyses, while several colleges 
lost points in any given year over the 2007-201 1 period of SAI implementation, no 
college lost points in total since the baseline year. Within any given year, there was 
significant variation in point growth, but the relative positions of the colleges remained 
stable. Colleges with high totals in the baseline year also had high totals in 2011. This 
applied to points growth as well. The proportion of points earned from each metric did 
not change much over time, with colleges accumulating approximately two thirds of their 
points evenly across three metrics: basic skills, college readiness, and 15 credits. The 
remaining points were accumulated via the other three metrics: 30 credits, quantitative 
reasoning, and degree completion. Although there was evidence of some gains in 
momentum for students who were already accumulating credits and making progress, 
overall student momentum does not seem to have changed much during the period in 
which the SAI has been in effect. 
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Research Question 3 

What distinguishes colleges that perform well according to the SAI framework from those 
that do less well? 

College enrollment size and total points earned are correlated, and enrollment size 
became a modestly more influential factor over time. Consistent with the SAI’s aim to 
encourage colleges to serve all students, including those from disadvantaged 
backgrounds, the basic skills metric appears to have encouraged enrollment from 
traditionally underserved groups. The characteristics of a college’s students — such as age, 
full-time enrollment status, race, or prior education — do not strongly influence total 
points or the change in points. There is little evidence that colleges serving more at-risk, 
low-income students are penalized by the SAI awards method. Based on the qualitative 
analysis of the interviews, it does not appear that colleges that perform well on SAI 
metrics are any more engaged in the implementation of SAI than colleges that perform 
less well; that is to say, SAI-related activities are not a distinguishing factor in 
performance. The relative performance of the colleges based on an absolute change in 
points — the metric by which funding is currently allocated to colleges — differs from that 
if the funding mechanism were based on other metrics of effectiveness and efficiency, 
including points per student and expenditures per point. 

Research Question 4 

Is the SAI Sustainable? 

As in 2010, we found more support among the colleges for the SAI framework 
and its metrics than for the funding mechanism. Support for the intermediate milestones 
of student progression represented in the SAI metrics seems to be somewhat higher than 
in 2010. The intermediate milestones are generally recognized as important things to 
measure and are seen as a helpful way to focus collective efforts on student progression, 
although many administrators and institutional researchers believe that to inform efforts 
to improve student success they need to be measured longitudinally rather than in cross- 
section, as is done for the funding allocation. Attitudes about the SAI funding mechanism 
are negative overall and have worsened over the last two years as SAI funding has come 
from reallocated base funds rather than as additional funds provided to the college system 
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as originally intended. The colleges generally expect the SAI — or some form of 
performance funding — to continue, given the national focus on accountability and college 
completion, and most are supportive of that vision of the future. And even though its 
independent impact is hard to isolate amid a larger collection of activities, most 
respondents seem to believe that the SAI, in the long run, will prove to have been a 
significant factor in increasing the focus on student achievement. 


3. Detailed Findings on Research Questions 


Research Question 1 

To what extent does the prospect of performance funding and the availability of 
performance data motivate colleges to implement policies and practices intended to 
improve student outcomes? 

To address this question, we asked interviewees a series of questions about the 
level of awareness and understanding of the SAI on campus, about its impact on the 
development of a culture of evidence and on college actions to improve student success, 
about the motivational effect of the funding, and about the level of collaboration among 
the colleges on issues of student progress and success. 

Awareness and understanding of the SAI. Despite being in its fifth full year of 
implementation, the SAI is not very well known across the colleges. We found little 
evidence indicating a spread of knowledge about the SAI since 2010. Substantive 
knowledge of the initiative is still mostly limited to top administrators down to the dean 
level; institutional researchers; faculty in basic skills, developmental education, and math; 
and faculty and staff that serve on committees overseeing activities related to student 
success. Others may have heard the name “SAI” or be familiar with the notion of 
“momentum points” or the “tipping point, ”“ but few know any details about the initiative 
or how their college has performed. 


2 These concepts arose from the research that led to the creation and implementation of the SAI. See 
Leinbach and Jenkins (2008) and Jenkins (2008). 



Local boards of trustees are more aware of the SAI than in 2010, which seems to 
be the result at least in part of the Governance Institutes on Student Success held the past 
two summers (see Box 2). 

Also, more colleges have 
incorporated SAI points, 
or measures similar to the 
SAI metrics, into their 
own performance 
monitoring reports and 
accreditation core themes 
than was the case in 2010, 
which has helped increase 
awareness of the initiative 
among local boards. 

Awareness of the SAI 
also seems to be up 
somewhat in the colleges 
that are new to the 
Achieving the Dream 
initiative (ATD; see Box 
3). This seems to be 
related to that fact that the 
SAI metrics are being 

used to monitor the performance of ATD-related activities as required by College Spark, 
which is providing funding for Washington colleges to participate in ATD. 

In general, there appears to be more awareness of an increased focus on student 
success and outcomes across the colleges, and of related activities and efforts. However, 
most of those interviewed do not attribute the increased focus to the SAI, believing that it 
reflects larger forces including the national completion agenda and an increased focus on 
student success among policymakers and funders in general and by the ATD initiative in 
particular. 


Box 2 

Governance Institute on Student Success (GISS) 

In 2011 and 2012, trustees and presidents from across the 
Washington State community and technical college system 
participated in institutes run by the Governance Institute for 
Student Success (GISS). A joint project between the Association 
of Community College Trustees (ACCT) and the Community 
College Leadership Program of The University of Texas at Austin, 
GISS institutes are designed to help boards of trustees and 
presidents develop the leadership tools they need to improve 
student success and completion. As part of the institutes, 
trustees and presidents examine data on students from their 
own institutions. 

For the June 2012 institute, members of the Research and 
Planning Commission (which is made up of institutional 
researchers from Washington two-year colleges) working with 
State Board staff, developed and ran for each college, using the 
SAI dataset, analyses of rates of quantitative course completion, 
completion of 30 college-level credits, and "tipping point" 
completion for cohorts of key student subgroups. This was a 
promising development since it produced for each college 
analyses that are more useful for benchmarking progress and 
completion outcomes for new student cohorts than are the 
cross-sectional SAI points measures used to award funding to 
colleges. 

Together with Achieving the Dream, the GISS has helped to 
increase understanding among college leaders of the value of 
longitudinal cohort analysis. 
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Box 3 

Achieving the Dream in Washington State 

In 2006, with funding from College Spark Washington, six two-year colleges in 
Washington State joined Achieving the Dream (ATD), a national initiative designed 
to improve outcomes for community college students. Achieving the Dream works 
with colleges to improve student outcomes by building a "culture of evidence" 
where decisions on how to teach and support students are made using data on 
what works. 

ATD provides each college with a leadership and data coach to help them with 
these efforts. ATD's approach is to use data on cohorts of students to track 
patterns of progression over time, identify opportunities for improving their 
progress and outcomes and evaluating the effects of improvement efforts. 

Encouraged by the accomplishments of the initial six ATD colleges in the state, in 
2011, College Spark Washington provided funding for ten additional Washington 
two-year colleges to join Achieving the Dream. As part of the grant, the ongoing 
evaluation of those additional ten colleges will be tied directly to the Student 
Achievement Initiative metrics. Colleges are required to develop their strategies in 
a way that aligns with the framework developed under the SAI. College Spark has 
tied all of its grant awards for community and technical colleges in Washington 
State to improving student achievement. 


We heard several explanations for the continuing low level of awareness and 
understanding of the SAI outside of central administration and basic skills: 

• The SAI is simply a low priority in many colleges in the context of 
budget cuts, increased workloads, and increased enrollments. The SAI 
gets lost in the midst of what are seen as larger and more immediate 
issues and priorities. 

• The ATD initiative overshadows the SAI in colleges that participate in 
it, because ATD allows colleges to focus on their own priorities, helps 
colleges examine student progression using longitudinal cohort data, 
and brings in coaches who provide help on understanding and making 
use of data. 

• The “need to know” about the SAI is seen as low for most people on 
campus, related to at least some leaders’ assessment that it is possible 
(and maybe preferable) to get people to focus on student success, and 
even work to increase student achievement based on the points 
represented by SAI milestones, without their having to know much, if 
anything, about the SAI. Leaders also minimize focus on the SAI out 
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of concern about possible “initiative overload” and related resistance 
from campus stakeholders. 

• Low awareness of the SAI among faculty is also related to a 

perception that the SAI is far removed from classroom activities and 
curricular issues. Faculty who do not receive SAI funding for special 
projects tend to see the SAI as not relevant to what they do. 

Box 4 contains selected quotations from our interviews about awareness and 
understanding of the SAI, reflecting views we heard from multiple respondents. 


Box 4 

Selected Quotations on Awareness of the SAI 

It doesn't seem like this initiative has really trickled down very 
much... I didn't hear anything about it as a faculty member. 
[Faculty/Division Chair, Business and Social Sciences] 

Around the campus there are so few people who understand SAI. I 
almost feel like I am starting at ground zero every single time I talk 
about it. [Institutional Research Director] 

We don't talk about SAI, and I can guarantee that most people don't 
even know what SAI is. [President] 

We have to articulate that information [about the SAI] to faculty and 
staff, and it is hard to do that unless you are one of the people on 
that statewide committee who really knows what the discussion and 
the intent is. [Dean, Workforce Education] 


Impact on developing a culture of evidence. The SAI is designed not only to 
motivate colleges to improve performance but also to provide them with data they can 
use to identify opportunities for improvement. In our field research, we perceived a 
disconnect between what people said in the interviews and what appeared to be the norms 
of practice at their colleges. Many respondents said that their colleges were developing a 
culture of evidence, likely reflecting the common wisdom in the postsecondary education 
arena that colleges should move toward data-driven decision making. But when pressed, 
few people could provide substantive examples. Our overall conclusion is that a culture 
of evidence is developing, but it is still in the early stages in most colleges. Use of data is 
still often limited to institutional top administrators, institutional researchers, and a few 
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“power users” on campus. Purposeful distribution of data is often limited to 
administrators and deans and to committees specifically working on issues of student 
success. There is no widespread use of data across the faculty at the 17 colleges we 
studied, although somewhat more use of data was reported for basic skills faculty, 
primarily related to federal reporting requirements. There continues to be much more use 
of traditional performance measures like course completion rates and graduation rates 
than of measures of intermediate progress like those included in the SAI. 

Where a culture of evidence is developing, it is only partly attributable to the SAI. 
ATD and other initiatives, as well as new accreditation requirements, were noted as 
having a greater influence on the use of data for planning and decision making. The GISS 
is widely recognized for introducing many college trustees to the value of tracking the 
progress of cohorts and key student subgroups within those cohorts. GISS analyses 
include intermediate measures of student progress that are also used in computing SAI 
achievement points, such as passing a college-level math course or earning 15 credits. 

The GISS measures are applied to cohorts of entering students to evaluate the 
forward progress (or “momentum”) of those particular students over time. This is in 
contrast to how the measures are used in the SAI, where colleges are awarded points for 
all enrolled students who reached those milestones during a particular year, irrespective 
of when particular students started or their momentum. SAI points as they are calculated 
under the initiative are not seen as useful by themselves, but colleges are beginning to use 
the SAI measures to track student success efforts by adapting them for longitudinal 
cohort analysis. In many cases, those aware of SAI data simply expect that the annual 
points will reflect what colleges do, rather than drive what they do. That is, it is assumed 
that the points “follow” from colleges’ general efforts to improve student success rather 
than being useful for designing or evaluating new strategies. 

A handful of colleges indicated that they do analyze the SAI points data, breaking 
down the points by demographic groups and/or program areas, but most colleges 
indicated that they simply review the points data provided by the State Board, without 
much additional analysis. Colleges that do work with the data often convert the total 
points to points per student, which they find more meaningful. The conversion removes 
the influence of enrollment size and changes in enrollment and is a better reflection of a 
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college’s success with individual students. SAI data related to basic skills and 
developmental education are used the most. Colleges see those data as more useful 
because activities to affect them can be attributed to particular units within the college 
(though we heard some criticism of the validity of CASAS test results, which are the 
basis of the basic skills points). Other SAI points, like earning 15 or 30 credits and 
certificate/degree completion, are more difficult to assign to particular units, giving 
faculty in other areas little reason to “own” the data. 

Some colleges noted that they have incorporated the SAI points into their broader 
performance management systems as additional measures or indicators of effectiveness in 
their annual fact books, institutional effectiveness scorecards, accreditation reports, or 
related documents. Also, several noted that measures similar to the SAI metrics (but 
applied to student cohorts tracked longitudinally) have been incorporated into grant 
proposals as additional measures of student outcomes, and may have increased colleges’ 
competitiveness for such grants. As already noted, College Spark intentionally requires 
colleges with second-round ATD grants to use such intermediate milestone measures. 

Box 5 contains selected quotations about SAI data and their impact on developing 
a culture of evidence. 


Box 5 

Selected Quotations on the Utility of SAI Data 

For us, this was a dataset that we didn't have access to before because 
we didn't have the capability. [President] 

Cohort-based ATD data drives everything. Everybody speaks that 
language, everybody understands Achieving the Dream. But nobody 
understands SAI because that is not how we measure ourselves. 
[President] 

The SAI data just don't tell the story that we need to know in order to 
make good decisions. They are just a snapshot. [Dean, Basic Skills] 

All we are looking for is the ability to apply student outcome data to 
decision making. As long as it is just outcomes thrown in a bucket, 
there is no way to make those connections. The cohort-based data 
overcomes that shortcoming. [VP, Instruction and Student Services] 
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Impact on college actions. Most colleges find it difficult to assess the 
independent impact of the SAI on their programs and activities. Most respondents across 
the colleges acknowledge a change in culture with an increased focus on student success 
in recent years, but most think that the SAI is only one (and not the primary one) of many 
forces pushing colleges to improve student success. Many can point to some general 
effects of the SAI, including providing a common language for discussing student 
progress and success. The SAI is frequently seen as having encouraged conversations 
across silos within a college, such as between student services and instruction or basic 
skills and college-level programs. It has encouraged a helpful focus on the student 
pathway and on transitions (e.g., basic skills to college programs, developmental 
education to college-level courses). 

In those colleges where the SAI money is allocated via grants for specific projects 
related to student success, the SAI is seen as encouraging innovation by providing 
funding for pilot efforts and professional development. Some college faculty spoke 
favorably about administrative support for innovations using SAI funds, including a 
greater acceptance that not all attempts at innovation will be successful. In providing 
examples of college actions related to student success, people often cited other influences 
including ATD, Title V grants, or other grant programs. Examples of specific actions 
attributed at least in part to the SAI included: 

• Changing post-testing procedures in basic skills to better capture skill 
gains (and points) 

• Redesigning basic skills programs to make them more structured like 
college-level courses, with attendance policies and grades 

• Implementing accelerated approaches to developmental education 
(e.g., modular approaches to pre-college math, reducing the number of 
levels of pre-college English) 

• Changing data processing/coding procedures (e.g., making a change in 
the numbering of courses that integrate math into technical coursework 
to get points for college-level math completion) 


3 Title V is a grant program of the U.S. Department of Education aimed at colleges designated as Hispanic- 
serving institutions. See http://www2.ed.gov/programs/idueshsi/index.html 
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• Better aligning curriculum or implementing new practices to better 
support student transitions from ABE to developmental education and 
college programs (e.g., introducing bridge courses or a transition 
coordinator staff position) 

• Increasing the focus on early supports for new students, including 
mandatory orientation and mandatory student success courses for 
particular groups of students. 

Box 6 contains selected quotations about the impact of the SAI on college actions: 


Box 6 

Selected Quotations on the Impact of the SAI on College Actions 

As far as driving institutional and classroom decisions about what to do with 
students, much more of that depends on what we have been able to generate 
ourselves rather than looking at SAI data. [President] 

SAI is merely something else that pushes us along, but we don't derive 
instructive detail from it. [President] 

SAI gives you an opportunity to start that conversation about just how hard it is 
for students to make a level gain in basic skills. That represents considerable 
effort. It allows for a more nuanced conversation about how— even though the 
majority of basic skills students don't transition to college— we are still adding 
value in the communities where we provide this service to people. [Dean Basic 
Skills] 

I don't think the points are telling the whole story. I don't know how, if you are 
not using cohort-based data, you can tell what the heck is going on. I don't 
believe that any college in the system is doing anything that moves those (SAI) 
numbers. I think it is demographics and the economy. [President] 

For the overall conversation (about student achievement), SAI is another tool. 
We have so many tools to determine if a student is achieving what they set out 
to do, and it is a tool to measure that. [IR Director] 


Motivational effect of funding. The SAI funding is not seen as much of a 
motivating factor for colleges, for several reasons: 

• The primary reason cited is that the amount is small. 

• The colleges have suffered substantial cuts in state funding despite 
sharp increases in enrollment. Concerns over how to make large cuts 
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while maintaining staffing and educational quality have overshadowed 
attention to what could be done to earn back a small share of base 
budget funding through the SAI. 

• Colleges cannot tell why their points go up or down, making it 
difficult to know what they would do differently if they were 
motivated to do so. 

• Funding is awarded “after the fact,” making it difficult to do any 
budget planning based on the money. Colleges, therefore, tend to 
assume they will not receive any SAI money rather than plan for any 
specific uses of it. 

• Some faculty and staff feel burned out and overworked due to budget 
cuts, and are defensive that they have always cared about student 
achievement and did not need SAI funding to encourage that. 

Where any colleges seemed to be motivated by the funding, despite the small 
amount, it reflected an assumption that the SAI will represent a larger share of funding in 
the future. 

Box 7 contains selected quotations about the motivational effect of SAI funding: 


Box 7 

Selected Quotations on the Motivational Effect of SAI Funding 

The intent of SAI is good and it is moving in the right direction; I just don't think we've put 
enough money behind it to really make significant change.... We still focus on student 
success and completion, so it's not like we're not doing that, but SAI would have more 
effect if there was more money behind it. [President] 

The incentive is so small that you could really say, "Oh forget about it; we don't need that 
$75,000 when you look at the big picture." But everyone knows that it's the right thing to 
do. That's why people work so hard— because it's the right thing to do, not because of the 
financial reward. [VP, Student Affairs] 

We don't even build it into our budget. We consider it a cut, and if it comes back to us we 
put it in the bank for a rainy day or an emergency. It's not predictable, it's not 
dependable, and it makes no sense. There is no logical way to know if you are going to get 
money from one year to the next. [President] 

It is mysterious ... something you see at the end of the year and then are glad there is not 
a lot of money involved. [Division Chair, Math/Science] 
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Collaboration among colleges. In the interviews we inquired about colleges’ 
perceptions of the assistance provided to them by the State Board and about whether the 
colleges are sharing strategies related to the SAI. In general, the SBCTC is seen as having 
provided good support in initially introducing and explaining the SAI, giving updates at 
various statewide meetings, and providing the SAI data to colleges (despite occasional 
concerns about timeliness of the data). However, colleges often noted that the State 
Board has not provided enough support on how to understand and make use of the SAI 
data and has not facilitated the sharing of effective practices across the colleges. 

Members of the Research and Planning Commission did acknowledge the State Board’s 
efforts to work with them to develop longitudinal cohort measures based on SAI metrics 
for the GISS. 

There is not a lot of sharing going on among the colleges in terms of strategies 
related to the SAI. What sharing that does occur is carried out informally rather than 
through any formal process or setting, for example, through phone calls or in side 
conversations at statewide meetings. The SAI is discussed in various WACTC 4 
commissions and councils, but discussion is mostly limited to general information and 
updates. Those forums have not been a source of detailed engagement in the SAI. The 
informal sharing of strategies that sometimes occurs across colleges is more generally 
related to student success rather than to the SAI specifically. Colleges are aware of which 
colleges are top performers in the SAI, and many report asking those colleges for help, 
but they also note that the inability of colleges to pinpoint the causes of point gains limits 
the value of sharing SAI strategies. 

The SAI is not seen as having affected the general level of collaboration across 
the colleges in the system. Many interviewees emphasized that the colleges are a fairly 
collaborative group that is willing to share, and the SAI has not done anything to reduce 
the willingness to collaborate. Some people thought that shifting a higher share of the 
base to fund the SAI in the future could increase the competitiveness among colleges. 


4 The Washington Association of Community and Technical Colleges (WACTC) is an organization of the 
presidents of the 34 colleges that develops policy recommendations for the State Board for Community 
and Technical Colleges. 
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Others disagreed, citing the strength of the collaborative culture among the Washington 
State two-year colleges. 

Research Question 2 

What is the range of performance in Washington ’s community and technical colleges 
according to the achievement point measures, and what are the trends in college 
performance over time? 

Our quantitative analyses examined the patterns of performance in earning 
achievement points at the 34 community and technical colleges during the period from 
2006-07 (AY 2007) through 2010-11 (AY 2011). They are based on detailed analysis 
conducted by CCRC (Belfield, 2012). Here we summarize the key findings on trends in 
the points earned by college, including the types of points earned. In addition, we 
examine whether there is a correlation between college performance and student 
“momentum” — that is, whether students in colleges that achieve greater points growth are 
progressing at a faster rate than students in colleges that do not do as well by the SAI 
metrics. 

Trends in points per college. 

Changes in total points. In the baseline year (academic year 2007), the average 
point total per college was 8,684 points. By 2009, the average rose to 10,365; by 2010, it 
was 11,598; and in 2011, it dropped slightly to 11,465. The college with the largest 
change in performance gained 7,556 points; the college that gained the least increased its 
total by 287 points. So, although several colleges lost points in any given year, no college 
lost points in total since the baseline year. 

Within any given year, there was significant variation in point growth. For 
example, in 2007-09, the total points generated by one college grew by considerably 
more than the point totals for any other college (+54 percent versus the next highest of 
+37 percent); four colleges grew by less than 10 percent. Over time, the gaps in point 
growth between colleges widened. From baseline to 201 1, the point total for the college 
that had the highest growth rate increased by 78 percent; all colleges gained points, but 
three gained less than 10 percent. At the same time, the relative positions of the colleges 
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remained stable. Colleges with high totals in the baseline year also had high totals in 
2011. This was a strong relationship and applied to points growth as well. 

Earlier analysis by CCRC suggested that between 2007 and 2009, the spread of 
total points among colleges was narrowing. In 2007, the college with the most points had 
5.7 times as many total points as the one with the fewest; by 2009, the ratio of most to 
least points fell to 4.4. By 201 1, this ratio rose again to 5.1. It is notable that most of the 
divergence among colleges in points and points growth took place primarily at the 
extremes, involving a few colleges. 

Changes in points by metric. Figure 1 shows how SAI points were distributed 
across each metric. At baseline, colleges were accumulating approximately two thirds of 
their points evenly across three metrics: basic skills, college readiness, and 15 credits. 
The remaining points were accumulated via the other three metrics: 30 credits, 
quantitative reasoning, and degree completion. These proportions did not change much 
over time; in 201 1, the proportion of points from each metric was almost exactly the 
same as in 2007. Until 2010, there appeared to be a trend toward increases in basic skills 
and college readiness points, but this was not sustained into 2011. 


Figure 1 

Points by Metric over Time 
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Table 1 reports the average points per college by metric. In the baseline year, the 
average college had 2,087 points from basic skills; by 2011, the average rose 38 percent 
to 2,872. Points from all other metrics increased. In fact, the average college saw college 
readiness and degree completion points rise the fastest between baseline and 201 1, at +46 
percent and +42 percent respectively. This suggests that achievement has risen. The slight 
decrease in total points between 2010 and 2011 was almost entirely a result of a decrease 
in basic skills points. 

However, this broad analysis masks within-college variations. For each metric, 
some colleges lost points and others experienced significant growth (as much as 50 
percent per year). In any given year, one third of colleges lost points in basic skills, 15 
credits, and quantitative reasoning; for the other metrics, one in five colleges lost points 
in any given year. Thus, in any year, colleges could easily lose points in a particular 
category. 


Table 1 

Points by Metric: Average Per College 


Metric 

Baseline 
AY 2007 

AY 2009 

AY 2010 

AY 2011 

% Change 
07-09 

% Change 
09-10 

% Change 
10-11 

% Change 
07-11 

Basic skills 

2,087 

2,788 

3,183 

2,872 

34 

14 

-10 

38 

College readiness 

1,811 

2,166 

2,580 

2,641 

20 

19 

2 

46 

15 credits 

1,777 

2,063 

2,172 

2,100 

16 

5 

-3 

18 

30 credits 

1,335 

1,538 

1,680 

1,662 

15 

9 

-1 

24 

Quantitative 

reasoning 

1,000 

1,059 

1,161 

1,229 

6 

10 

6 

23 

Degree 

completion 

674 

751 

822 

961 

11 

9 

17 

43 

Average per 
college 

8,684 

10,365 

11,598 

11,465 

19 

12 

-1 

31 


Overall, there are more opportunities for colleges to earn basic skills and pre- 
college points. In contrast, the growth in points for the accumulation of 15 or 30 college 
credits was less dramatic. This disparity makes sense because the basic skills and college 
readiness metrics require students to improve their scores on a basic skills test or progress 
through a sequence of remedial courses rather than take and pass a college-level math 
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course or accumulate college credits that count toward completion of a credential. A few 
colleges did increase their degree completion points. On this metric, only one college 
accumulated fewer points in 201 1 than at baseline, with a 4 percent decrease. Overall, 
these trends have led to shifts in how some colleges accumulate points, though the 
ultimate effect is not dramatic. Colleges that grew faster tended to start from a lower 
base, so they converged to the average. 

Evidence of increases in student momentum. An overarching goal of the 
Student Achievement Initiative is to motivate and guide colleges to undertake changes in 
practice that lead to improved student outcomes. Indeed, the first principle is that the 
initiative should lead to improved educational attainment for students, with a specific 
goal of reaching the “tipping point” (of at least one year of college credits and a 
certificate) and beyond. In the years since the SAI began, the State Board and colleges 
have discussed how to increase student “momentum,” that is, how to help more students 
progress further and to do so faster. We analyzed the data from the SAI to see if there is 
evidence that student momentum has increased since the SAI was established. 

To do this, we examined the progression of students along a pathway. Students 
with no prior college enter with no points. If they are assessed to be academically 
unprepared for college-level work, they could first earn basic skills or college-readiness 
points. Those who enter college ready or who progress from remedial instruction can earn 
points by earning 15 and then 30 college credits or by passing a college-level math 
course. Students at the end of the pathway get points by earning a credential or 
completing an apprenticeship program. “Momentum” refers to students’ progress along 
this pathway of points. 

To measure student momentum, we tracked the progress of each student from one 
year into the following year to see how many points were accumulated. We disaggregated 
these analyses for subgroups of students categorized as basic skills, transfer, or workforce 
students, following definitions used by the SBCTC to classify students by the type of 
program in which they are enrolled. The tracks are based on each student’s status in the 
first year. It is possible to see, for example, how many of the college-ready students in 
2007 completed 15 credits in 2008. Rates of progress can then be compared across years 
to see if colleges are generating more momentum along a pathway for a given set of 
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college-ready students. As noted above, the pathway of progression in points is: none, 
basic skills, college readiness, 15 credits, 30 credits, quantitative reasoning, and 
completion. Basic skills students can earn any category of points. Transfer students and 
workforce students typically do not earn basic skills points but can earn college readiness 
points if they enroll in and complete college remedial or “developmental” courses. 
Figures 2 through 4 illustrate momentum patterns for students from 2009 to 2010 based 
on how many points and what types they started with in 2009. 

Figure 2 shows that half of students who started with no points in 2009 (50 
percent of students in that year) had not earned any points by the end of 2010. In fact, 
looking over three years, few basic skills students went on to earn any college-level 
points. Most of these students who did not earn points in either year were not enrolled in 
2010; this is one way that students can lack momentum. This was especially true of basic 
skills students. Regardless of the category they started in, very few students who 
accumulated no points in the first year went on to accumulate any points over three years. 

As is evident from Figure 3, students who started 2009 with college readiness 
points — indicating that they had previously taken and made some progress in adult basic 
skills or developmental courses — made more progress. Over 20 percent of transfer 
students who began with college readiness points, more than 25 percent of basic skills 
students, and nearly 35 percent of workforce students attained at least one additional 
achievement point between 2009 and 2010. Interestingly, other analysis not shown here 
indicates that the momentum of students in either transfer or workforce programs did not 
seem to depend on whether students first took pre-college remedial courses or entered 
directly into college-level programs. 

Students who started 2009 with 15 college credits progressed even further (Figure 
4). At least a third of basic skills students who began with at least 15 credits and over 40 
percent of transfer and workforce students progressed along the pathway between 2009 
and 2010. Generally students who were further along the pathway exhibited greater 
momentum. 
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Figure 2 

Distribution of Points by end of 2010 for 
Cohort of Students Who Started Zero Points in 2009 



Figure 3 

Distribution of Points by End of 2010 for Cohort of Students 
Who Started with College Readiness Points in 2009 



■ Certification 

■ Quants 

■ Credit 30 

■ Credit 15 

■ College ready 
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Figure 4 

Distribution of Points at End of 2010 for Cohort of Students 
Who Started with 15 Credit Points in 2009 



Not surprisingly, attrition rates were much lower for students who already had 
points. However, since these students represent less than 40 percent of all students, it is 
difficult for an improvement in their momentum to significantly increase the total points 
a college can obtain. Since half of students did not accumulate any points and few basic 
skills students go on to earn college-level points, despite some gains in momentum for 
students who were already accumulating credits and making progress, overall student 
momentum does not seem to have changed much during the period in which the SAI was 
introduced. Put another way, improvements in momentum by the smaller number of 
students who were already making progress had a small effect on points accumulation 
overall. Looking back at Figure 1, the proportion of points that are accumulated toward 
the beginning of the pathway — basic skills or college readiness points — remained stable. 
If colleges were increasing momentum, a greater proportion of their points in later years 
would have been gained from the metrics that take place later in students’ academic 
pathways (e.g., 30 credits and degree completion). 

Ironically, colleges that retain more students may actually find it harder to 
accumulate points for those students beyond their first year — simply because there are 
fewer points for students to earn beyond the first year. Given the large numbers of 
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students who earn no points, colleges may be more likely to improve student momentum 
if they focus on the gains that can be made among students when they first enter the 
institution. This would suggest efforts to improve student attachment in their first year. 

Research Question 3 

What distinguishes colleges that perform well according to the SAI framework from those 
that do less well? 

To examine the factors that distinguish colleges that perform well according to the 
SAI framework from those that do less well, we analyzed factors such as college and 
student characteristics that correlate with points increases. Based on our fieldwork, we 
examined if there is a correlation between responses about the SAI and college 
performance — that is, were high-performing colleges more engaged in implementing the 
SAI, or did they have different views about the policy than low-performing colleges? In 
addition to looking at the determinants of and responses to institutional performance on 
the SAI, we compared college performance according to change in total points from a 
prior year — the metric used to award funding under the current policy — with other 
alternative performance metrics, such as points per student, to see if different metrics 
would result in a different funding allocation. 

Correlates of college performance. 

Effects of student characteristics. If the SAI is working as intended, the 
characteristics of a college’s students should have little or no influence on college 
performance. That is, the SAI policy should not pressure colleges to serve students they 
would not otherwise seek to serve based on their missions. To assess this, we examined 
the correlation between a wide range of student characteristics and college performance 
according to the SAI measures. 

Our analysis found that many of the student characteristics we examined were 
correlated with SAI performance. For example, full-time students generated more points 
than part-time students. This is not surprising since full-time students generally make 
faster progress than part-timers Students on financial aid also tended to accumulate more 
points. This is also not surprising because to receive financial aid students must be 
pursuing a credential, whereas some students not on financial aid may just be taking a 
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few courses and not intending to pursue a credential. Students under 20 were much more 
likely to earn points compared to other age groups, and students over 65 were much less 
likely. Among full-time students, males, high school dropouts, and Black students were 
less likely to accumulate points. Both high school dropouts and high school graduates 
were more likely to gain points as part-time students. Again, none of these findings are 
surprising. 

While most of the student characteristics we examined had a statistically 
significant association with points accumulation, the apparent effects of these 
characteristics were small, explaining between 2 and 10 percent of the variation in points. 
Moreover, there was considerable variation in effects across the individual measures; that 
is, different characteristics affected different measures differently. The result was that in 
the aggregate, no particular characteristic affected points overall in a substantial way. 
Thus, no particular college had an easier way to accumulate total points based on the 
characteristics of the students it serves. In particular, there is little evidence that colleges 
serving more at-risk, low-income students are penalized by the SAI awards method. 

Not surprisingly, colleges serving more disadvantaged students were more likely 
to earn basic skills points. Put differently, the basic skills points do seem to reward 
colleges for serving disadvantaged students. This is consistent with the SAI’s aim to 
encourage colleges to serve students across mission areas, including those from 
disadvantaged backgrounds. For these colleges, however, the gains in basic skills points 
were counterbalanced by lower points accumulation in the college-level metrics. Serving 
disadvantaged students in basic skills programs did not give colleges an advantage in 
terms of the overall SAI points. 

Influence of college size. Both total points per college and change in total points 
were strongly related to size of enrollment, or headcount. This relationship holds even 
after controlling for the composition of the student body in terms of socioeconomic 
status, age, full-time/part-time enrollment, disability, race, and prior education. The 
points-headcount relationship was reasonably consistent across all three years we 
analyzed. Larger colleges saw greater increases in total points. A college with 1,000 more 
students in the 2007 baseline year would have accumulated 165 points more than the 
average college by 2011. According to State Board staff, this is consistent with the 
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original design of the SAI: Larger colleges were expected to be able to obtain bigger 
awards on the premise that they would require more resources to make changes. The 
influence of size has increased over time, but the trend is not dramatic. Few other student 
body characteristics were positively associated with total points or points growth in a 
consistent way. Thus, point totals were not driven by characteristics of the colleges, such 
as the composition of the student body. This conclusion holds generally for each of the 
six metrics. 

Relationship between college performance and responses about the SAI. 

Based on the qualitative analysis of the interviews, it does not appear that colleges that 
perform well on SAI metrics are any more engaged in the implementation of SAI than 
colleges that perform less well. SAI-related activities are thus not a distinguishing factor 
in performance. We looked for any general patterns relative to performance by 
comparing the group of higher performing colleges among those interviewed (those with 
an increase in points of more than 4,000 from 2007 to 201 1) to the group of lower 
performing colleges (point increases of less than 2,000). There is not much difference 
between high and low performers in the level of awareness; in both groups, the SAI is 
primarily known among administrators down to the dean level and faculty/staff who 
serve on relevant committees. Neither group of colleges reported that the SAI had much 
direct impact on their activities. Where activities were cited as being at least in part 
attributable to the SAI, they tended to be in the area of basic skills and developmental 
education at both sets of colleges. 

The one area with some detectable differences between higher and lower 
performing colleges was in attitudes about the SAI, with attitudes appearing to be 
somewhat more negative in the lower performing colleges. While most of the lower 
performing colleges noted that the SAI measures represent important milestones and can 
focus conversations on student progress, they more often noted concerns including issues 
with the timeliness and reliability of the data, sensitivity of the metrics to enrollment 
changes, the cross-sectional (as opposed to longitudinal) nature of the measures, and the 
lack of applicability to some unique characteristic of their college structure or student 
population (e.g., primarily transfer-seeking students or those intending to take only a few 
courses to upgrade job skills). Negative opinions about aspects of the funding mechanism 
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were common to both groups of colleges, but were more pronounced in the lower 
performing colleges. While high performers may have been happier with the outcomes, 
they were uniformly quick to acknowledge that they did not truly understand why they 
had scored relatively well in point gains. 

Alternative measures of “achievement.” The SAI is explicitly intended not to 
rank colleges, and in general ordinal ranking is often not the best way to understand 
differences between colleges (Jenkins & Cho, 2012). The SAI rewards colleges based on 
their improvement against their own historical baselines, not against the performances of 
other colleges. It is a funding system, not an ordering system. Still, performance systems 
should be robust — they should be insensitive to variations in how they are calculated. 
Moreover, the SAI does allocate funds based on colleges’ relative performance. Some 
colleges receive more funds than others. Thus, the allocation system of the SAI will be 
more credible if it reflects a valid measure of relative performance. Colleges that receive 
more funds using the SAI measure of achievement should also receive more funds based 
on an alternative, equally plausible measure. 

To examine this we compared the rankings of colleges over time using the SAI 
measure and using alternative measures of “achievement.” (The alternative measures do 
not alter the specific metrics by which students generate points for colleges; rather they 
are alternatives to the way that those points are ultimately combined to produce a 
measure of college achievement.) There are multiple ways that colleges’ relative 
performance within the Washington State system could be measured. In this analysis we 
compared the SAI measure — which is change in total points compared to a baseline 
year — to three other measures: points per student (captured both in terms of headcount 
and FTE), change in points per student, and expenditures per point. Points per student is a 
measure of effectiveness, since it examines the points earned by colleges independent of 
their size. In our interviews with colleges we found that some colleges recognized this 
and were converting the SAI points to points per student as a measure of their 
effectiveness. We examined points per student in a given year as well as changes in 
points per student from a baseline year. The third alternative measure — expenditures per 
point — is a measure of the efficiency with which colleges invest their resources in order 
to achieve points. This is admittedly a rough measure, given that the data used for 
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expenditures are drawn from IPEDS and are sometimes criticized as inaccurate by 
college finance experts. 

As in a prior analysis (Belfield, 2012), in this more recent comparison, using the 
2009 and 2010 data (the latest available), we found that the ranking systems based on 
absolute change in points yields a different ordering of colleges than rankings based on 
any of the other three measures. Some of the colleges that are ranked very low based on 
absolute change in total points are ranked high when the ranks are based on any one of 
the three alternative measures. The correlations between the SAI measure and the other 
three are negative. This suggests that the SAI measure is measuring something different 
than the other measures. We can assume therefore that the factors associated with 
improving performance on the SAI measure are likely different than those associated 
with improving performance on the alternative three measures. 

Research Question 4 

Is the SAI sustainable? 

To address this question, we asked interviewees about their attitudes toward the 
SAI policy and changes in those attitudes over time, about the degree of 
institutionalization of the SAI within the colleges, and about their perceptions of the 
future prospects of the SAI. 

Attitudes about the SAI. There are two kinds of attitudes about the initiative that 
are important in our study — attitudes about the framework of metrics on which the points 
are based, and attitudes about the funding mechanism by which rewards are allocated to 
colleges. As in our 2010 research, we found more support for the framework and its 
metrics than for the funding. 

Support for the intermediate milestones represented in the SAI metrics seems to 
be somewhat higher than in 2010. They are generally recognized as important things to 
measure and are seen as a helpful way to focus collective efforts on student progression. 
The higher support is likely related to the measures being integrated into other initiatives 
and student success efforts at the colleges (such as ATD) and their being used to measure 
the impact of those actions. It is important to note however that the colleges are generally 
applying measures similar to the SAI metrics to longitudinal cohorts of students. The fact 
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that the measures were based on research providing evidence of their importance 
continues to be widely appreciated across the colleges. 

The framework’s focus on underprepared students is widely believed to have 
increased the visibility and priority of the basic skills mission of the colleges and to have 
focused attention on the importance of transition from Adult Basic Education (ABE) to 
college-level programs. Basic skills faculty and administrators in several colleges pointed 
out that the SAI’s point structure may have helped insulate basic skills departments from 
worse budget cuts, given that basic skills students pay no tuition, by allowing their 
departments to generate some funding for colleges. However, an example from one 
college demonstrates that the SAI’s focus on basic skills did not have that effect on all 
campuses. The college chose to cut selected basic skills programs in order to avoid cuts 
in college credit programs, despite recognizing that the outcome would be a loss in SAI 
points. The loss in points resulted in the college receiving no SAI funding for the year, 
but administrators noted that the loss in SAI funds was substantially less than what they 
would have lost in tuition and other revenue by cutting credit programs. Moreover, the 
college made clear that the cuts it made to its basic skills program did not reflect a lack of 
commitment to that mission. In fact, the college decided to close some satellite basic 
skills centers located throughout its service region, which generally had poor outcomes, 
and to strengthen its on-campus basic skills instruction. 

We heard fewer concerns about the specific metrics adopted by the SAI than we 
did in 2010, possibly due to awareness that the SBCTC was in the process of considering 
some changes to the measures during our interview process. Also, respondents seemed to 
be more focused on the bigger picture (i.e., the value of tracking student progression) and 
to be less concerned with the details of specific metrics. On the other hand, we more 
often heard general concerns about the SAI approach to using the metrics and data 
including: 

• The cross-sectional nature of the data severely limits its value for 
identifying strategies to improve student progression or tracking the 
effects of such strategies. 

• It is difficult to understand why SAI points go up or down because the 
points cannot be connected to particular college actions. 
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• The requirement to demonstrate year-to-year improvements to receive 
funding disadvantages colleges that were higher performing to begin 
with; conversely, the SAI can provide larger rewards to colleges with 
poor initial performance. 

• The major impact of enrollment on points limits the extent to which 
the SAI is really funding “performance” in improving student success. 

• Coding differences and data anomalies across colleges also have a 
large impact on SAI points (e.g., how courses are coded can affect 
whether points are earned for completion of a college-level math 
course, as in the case of a course that integrates the teaching of 
college-level math skills into a technical course that is not coded as a 
“math” course). 

While attitudes about the SAI framework and metrics, overall, continue to be 
largely positive and even somewhat improved, attitudes about the funding mechanism are 
negative overall and have worsened over the last two years. The most frequent concern 
we heard about the funding was about the “skimming” of base funds to create the SAI 
funding pool, which is viewed as reneging on the initial idea that the SAI be funded with 
“new money,” although the SBCTC is not typically blamed for this development. 

Despite the negative opinions about the SAI funding mechanism, we found more 
support for the general concept of performance funding than in 2010. Some of that is 
resigned support because people believe performance funding in some form is here to 
stay, and they want to make it work. Many of those voicing these sentiments understand 
that performance funding is popular with the legislature and governor and that the SAI 
may be protecting the system from even larger budget cuts. But some respondents 
expressed more active support for performance funding, arguing that “money talks” and 
that more money allocated based on performance would be more effective. We heard two 
very different perspectives on why more money is better. Some respondents suggested 
that more money would buy more projects and pilot programs, a perspective generally 
held at colleges that allocate SAI funds through targeted “mini-grants” or special 
allocations, as well as among recipients of such funds. Others noted that it would simply 
be a stronger motivation if a larger share of the total budget for the college system was 
based on performance. However, some of those who argued that allocating more of the 
budget based on performance would likely provide a stronger motivator for change were 
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also nervous at the prospect, fearing that lower performing colleges could lose funding 
and thus have fewer resources with which to improve. 

Box 7 contains selected quotations about attitudes toward the SAI metrics 
framework and funding. 

Box 7 

Selected Quotations on Attitudes About SAI Metrics and Funding 

The language has been helpful, the idea of "moving the needle" [on student achievement]. 

Those kinds of things have become the normal discussion and language in all our 
discussions. ..[whereas] it seemed like all of the discussions back in the day used to be about FTE 
and generating FTE. Now all the discussion is about retention, completion, progression. It has 
flipped that, and I think it was a good effort. [President] 

Fundamentally, it is a relatively simple methodology to ostensibly measure some very complex 
variables. That alone should raise several flags. [VP, Institutional Effectiveness] 

We were given an opportunity to show our value and keep our jobs. [Dean, Basic Skills] 

It doesn't measure our primary goal of putting people to work. [Faculty Union Leader] 

This was going to be used initially as a way of allocating new funding, and we would be 
measured only against ourselves. But now that it has become the basis for reallocating existing 
funds, I continue to have very strong and persistent worries about using this as a funding 
mechanism. I don't think it necessarily promoted the kind of behavior that we want. [President] 

Really large colleges have an advantage over small colleges. It was not supposed to be an 
enterprise that pitted resources of one college against another. That is one of the 
disappointments. The size of the institution does make a difference because of the larger 
numbers in each of these measures. For this to happen during the worst economic times that 
we have ever faced in higher education, and every college tried to respond to those budget cuts 
as fit their abilities. [President] 

As we look at the landscape around us, many of the states are moving toward some percentage 
of funding based on student success. SAI has provided us with a framework with some data 
points that are comparable with other colleges, so it gives us a way to benchmark our progress. 
[President] 


Management and institutionalization of the SAI. In about a quarter of the 20 
colleges where we conducted interviews, some initial effort was made to have a separate 
SAI committee or task force to oversee efforts related to the initiative. Most of those 
colleges integrated it with related initiatives overseen by "retention" or "institutional 
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effectiveness" or ATD committees and administrators. In about half of the colleges, some 
effort was made to incorporate attention to the SAI into existing committees or structures 
from the beginning, and that remains the case. As noted, the SAI has been intentionally 
incorporated into the ATD expansion for the newest round of colleges funded by College 
Spark. In the remaining colleges (about one quarter) there was never any concerted 
attention devoted to management or implementation of the SAI, and it remains largely 
marginalized, with senior administrators and IR directors dealing with the SAI only to 
comply with State Board directives or to participate in system-wide discussions. 

Some colleges have added SAI points or cohort-based measures similar to the SAI 
metrics to their strategic or institutional effectiveness plans and have built them into 
accreditation core themes and campus enrollment management plans. There have been no 
major changes since 2010 on who is involved — it is still very limited to top 
administration and faculty/staff in selected units. There continues to be a mix of 
approaches to the use of SAI funds. About a third of colleges we interviewed put the 
funds in their base budgets, and the rest use the funds for small grants and targeted 
projects that they see as related to student achievement. In summary, there has not been 
much change in the scope and scale of involvement in or management of the SAI at the 
colleges, although the initiative has been somewhat more integrated into planning, 
institutional effectiveness, and accountability processes in some colleges. 

Future prospects for SAI. Opinions are mixed about whether the SAI is likely to 
have much impact on improving student success, with few seeing an independent impact 
but most believing that the SAI is one factor among others contributing to more focus on 
student success. Many appreciate that the SAI provides some new metrics related to 
student success and think that that could help to improve outcomes over time, particularly 
if converted to longitudinal cohort measures. Most colleges expect the SAI to continue, 
or at least for some version of performance funding to continue. The national attention 
the initiative has generated and the commitment of the State Board and Washington’s 
legislature are seen as important factors supporting its continuation. The initiative’s 
survival through recent budget cuts is seen as evidence of that commitment. 

Acceptance that some funds should be awarded to colleges based on student 
progress and outcomes is fairly widespread, and many favor some more money to that 
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end. But many express concern about it becoming a large share of funding on the grounds 
that so much about students’ challenges and reasons for not persisting is beyond the 
control of colleges. Many colleges have heard that some changes to the metrics are 
currently being considered by the State Board and are likely to occur, though specifics 
are not widely known. Those familiar with the details of the recommended changes are 
generally supportive of them. 

Some of the more common suggestions respondents offered for improving SAI 
include: 

• Emphasize the use of student progression metrics in cohort data, rather 
than cross sectional data, to better allow for connection to changes in 
college activities/efforts 

• Use points per student rather than total points 

• Measure changes in “momentum” from one year to the next 

• Measure progress within a specified period of time to award points, 
like completion of 15 or 30 credits within one year of entry 

• Measure progress and award points within specific groups of students 
(e.g., those seeking a degree, those beginning in developmental 
education) 

• Incorporate additional credit accumulation points beyond 30 credits 

• Award points for transfer and transfer readiness 

• Award points for short-term certificates 

• Measure and award points for workforce outcomes for CTE students, 
like finding a job or a better job, or increasing wages (though 
measurement challenges were acknowledged) 

• Provide colleges with more help in interpreting the data and analyzing 
what strategies work to increase student success. 
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Box 8 contains selected quotations about the future prospects for the SAI. 


Box 8 

Selected Quotations on Future Prospects for the SAI 

A lot of people are resigned to the fact that performance funding is our 

future. ..We want this to be a tool that helps us and doesn't hurt us. [VP, Student 

Affairs] 

The state board loves it, the public seems to think it is a good idea, our legislators 
think it is wonderful, so I think we will have something like it for a long time. It is 
part of the fabric of higher ed. [President] 

We have slipped into trying to please everyone. It feels like it may be getting 
watered down a bit because we are trying to please various constituents. If 
student achievement is about progression, then why are we continuing to look at 
things that do not move students toward completion [regarding statewide 
advisory group's efforts to revise SAI]? [Administrator] 
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4. Conclusion 


The Student Achievement Initiative has garnered much attention, and the State 
Board much admiration, in part for how thoughtfully the program was developed and 
designed. The process began with a broadly representative group developing principles 
that then served to guide the more specific design and implementation of the initiative. At 
this five-year point of implementation, as the State Board is reviewing the past 
experience and considering some modifications to the initiative, it is appropriate to 
conclude with our reflections about how the SAI has played out with respect to the initial 
twelve design principles. 

Overall Principles 


Overall Principles 

• The initiative leads to improved educational attainment for students, specifically 
the goal of reaching the "tipping point" and beyond. 

• The initiative allows colleges sufficient flexibility to improve student achievement 
according to their local needs. 

• The initiative results in the identification and implementation of successful 
practices to improve student achievement system-wide. 

Student achievement, as measured by the achievement point metrics, has 
increased significantly since the advent of the initiative. However, our analysis has 
revealed that individual student momentum, or the progression of students toward the 
tipping point and beyond, has not increased overall in parallel with achievement points. 
While we find some evidence of increased momentum among students who were already 
making progress in college-level programs, about half of students in any given year do 
not accumulate any points, and very few adult basic skills students go on to earn any 
college-level points. Total points and change in total points are both correlated with 
college size, meaning that larger colleges earned more awards, a trend that increased 
somewhat during the first five years of the initiative. This finding illustrates an important 
issue related to the initial design of the SAI: A cross sectional measure of points earned 
collectively by all students enrolled in a particular year is quite different from measures 
of student progression over time. There was widespread acknowledgement that if 
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colleges are to identify and measure the impact of strategies for improving student 
outcomes, they need to use longitudinal cohort data rather cross-sectional “snapshots.” 
As implemented, the SAI has honored the design principle of allowing colleges 
flexibility to improve student achievement according to their local needs. There have 
been no prescribed activities, as colleges may decide which area(s) of the progression 
framework they want to focus on and how they will try to improve student success. 
However, the other side of the flexibility coin is that many faculty and staff are seeking 
more guidance in the identification of successful practices. We heard in nearly every set 
of interviews that it has not been possible to understand what actions have led to 
increases in SAI points. This uncertainty has limited the identification and sharing of 
effective practices. 

Principles for Measurement 


Principles for Measurement 

• Performance measures recognize students in all mission areas and reflect the 
needs of the diverse communities served by colleges. 

• Performance measures must measure incremental gains in students' educational 
progress irrespective of mission area. 

• Measures are simple, understandable, reliable and valid points in students' 
educational progress. 

• Measures focus on student achievement improvements that can be influenced by 
colleges. 

This set of design principles has been quite well satisfied in the implementation of 
the SAI. We found strong support for most of the metrics and an appreciation that they 
were largely valid measures derived from research . 5 Interviewees recognize that there is a 
tradeoff between simplicity and full accommodation of all aspects of the mission that are 
felt to be important, but there was a growing sense over the course of our three-year 
evaluation that the “back end” of the progression framework could be better rewarded as 
there are considerably more opportunities to earn points in the pre-college portion of the 
point framework than in the later stages of student progression. At the same time, we did 
hear from some faculty who teach transfer and higher level courses that the means by 


5 CCRC’s research findings, used in part to inform the SAI metrics, were based on longitudinal data 
tracking cohorts of students, not on analyses of cross-sectional data. 
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which they could influence gains in college-level achievement points were far less clear 
to them than how pre-college points can be increased. Overall, however, the SAI has 
faithfully adhered to the measurement design principles. 

Principles for Incentive Funding 

Principles for Funding 

• Colleges are rewarded for improvements in student achievement. 

• Funding is structured so that colleges compete against themselves for continuous 
improvement rather than competing with each other. 

• Funding is stable and predictable, and cumulative over time. 

• Incentive funding rewards student success and becomes a resource for adopting 
and expanding practices leading to further success. 

• New funds provide the greatest incentive. 

This set of design principles has proved more challenging during the 
implementation of the SAI, for a number of reasons. Most significantly, the reductions in 
state support to the system resulted in the reallocation of base funding to sustain the 
initiative instead of the “new funds” that were envisioned. This violated not only the 
design principles but the trust of many across the colleges who expressed strong 
resentment about receiving “rewards” that could be less than the amount taken from their 
base allocation to put toward SAI awards. 

Contributing to the challenge with these principles were some initial 
misconceptions and, we believe, shortcomings in the principles themselves. New funds 
are certainly more politically palatable to colleges than reallocated funds, but it is not 
clear that they provide the greatest incentive. An incentive funding system designed at the 
outset to add incrementally to an incentive fund in part by reallocating from the system’s 
base budget might have provided strong incentives for systemic change. But having 
conditioned the colleges to expect only new money and to expect to use those funds “as a 
resource for adopting and expanding practices” seems to have created a strong 
expectation that SAI dollars were to be used to fund specific projects or activities rather 
than to create more fundamental shifts in institutional priorities toward student 
achievement. When these funds had to be used instead of offset (sometimes only 
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partially) the reductions made to base budgets, colleges found it increasingly difficult to 
support the new activities they associated with the Student Achievement Initiative. 

A second area of some confusion concerns the principle about colleges competing 
against only themselves. Clearly the computation of year-to-year point gains was 
consistent with the principle, but when it came to ranking the colleges according to these 
year-to-year gains and, more significantly, when it came to redistributing funds via SAI 
point gains, colleges found themselves in competition with one another. Even in better 
fiscal times there is a fixed amount of funds to allocate based on the relative performance 
of colleges, so the original principle, while understandable in a system that justifiably 
prides itself on collaboration, likely introduced some unrealistic expectations. 

The final product of this three-year research project will be a policy brief in which 
we revisit, with the benefit of two more years of quantitative and qualitative data, the key 
issues that the State Board has been facing in the design and implementation of the 
Student Achievement Initiative. The Board continues to face difficult decisions about 
how to define, measure, and reward success as it works with its colleges to reconfigure 
the initiative to adjust to new, starker fiscal conditions. Other states have already learned 
much from the State Board about how to engage thoughtfully with constituent colleges in 
designing performance funding systems. The Board will shortly make some significant 
changes to the SAI measurement and funding systems, based on careful review of 
available data. This should yield more lessons for states that are striving to improve 
student achievement in a resource-constrained environment. 
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